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Attention. of research and evaluation specialists has 
ofxen focused on methodology for assessing program outcomes vith less 
attention given to techniques for describing important 
characteristics of a program or methods for assessing the degree to 
vhich a program may faithfully reflect a particular model. The 
purposes of this paper are to describe a rationale for assessing the 
degree of program implementation, to identify some common 
misperceptions regarding program fidelity, and to discuss approaches 
used in assessing the degree of program implementation of a 
large-scale career education program called Experience-Based Career 
Education (EBCE) . The authors' experiences in developing and using an 
EBCE Essential Characteristics Checklist and an EBCE Process 
Checklist in pilot sites in four states are described. Attention is 
also given to alternative uses of instruments that assess program 
implementatioB that can be made by program staff and by eyaluators. 
(Author) 
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ASSESSING THE LEVEL OF IMPLEMErJTATION 

OF NEW PROGRAMS 

Thomas R. Ov/ens and Joseph F. Haenn 
Northwest Regional Educational Laboratory 



In the gonoral educational program evaluation paradigm, the focus is on 
desirable educational outcome measures. These outcomes are then attributed to 
certain program design elements / although the linkages are rarely explicit. 
However/ as Charters and Jones (1973) state, 

Whiat is not standard practice in evaluation studies is to describe, 
Ifit alone measure, how the programs in "experimental" and "control" 
situations actually differ from one another — or even to certify that 
tl\ -j: do. A serious consequence of slighting • the alleged "independent 
v.-iri.-'ble" is that elaborately designed evaluation studies may 
sor.-- *.i !ues end up appraising non-events, with no one the wiser, (p. 5) 

Moasuroment of the independent variable in implementation studies is 
generally referred to as "degree of implementation" or "measurement of program 
fido li ty . " Those concepts are useful in reviewing progrcim or curriculum 
iippl^'nentation efforts. It is not the purpose of this paper to review the 
Ittor-iture on progr^ua implementation (see Fullan and Pomfret, 19 75, P£iul , 

197o, for reviev/s of implementation research) . However, it is important to 
consid/jr sorae de ^'ined features v/hich may affect the implementation process. 

?.'-.'j-jrs and Shoemoj<or (1971) have identified five characteristics v;hich may 
iri fiu::ri ..'J the implementation of innovations: 

Palac^ve Advantage: The degree to which the innovation is perceived 

ar, boing better than the idea it supersedes. 
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Conipa tab 1 1 i t y : 



The degree to which the innovation is perceived 
a:i consistent v/ith existing values, past 
Os :)eriences / and needs. 



Complexity : 



The degree to v/hich the innovation is perceived 
as relc'itively difficult to understand cind use. 



Trialability : 



The degree to which an innovation may be 
experimented with on a limited basis. 



Observability: 



The degree to which the results of cin innovation 
are visible to others. 



These characteristics generally influence a decision to accept or reject an 
innovation. Once a decision to accept — or at least attempt to implement — axi 
innovation has beeh made, it is important to measure the actual extent of implementa- 
tion in order to permit * causative statements about program effects. The literature 
is rich with examples of innovations which either have not been implemented to 
the degree intended/ or have not been implemented at all in practice. By 
measuring what has actually been implemented, one can relate program outcomes to 
program features. Evaluation of this type corresponds to process evaluation in 
Stuff lebeam* s (1971) CIPP evaluation model, cind to program implementation in 
Alkin's (1969) evaluation theory. 

Methods of Measuring Degree of Implementation 

Fullcin and Pomfret (1975) have identified three elements often considered in 
developing a degree of implementation instrument: structure, behaviors and knowledge. 
Structural features include those changes in formal arraingements and physical 
conditions defined by the developer as necessary for program implementation — 
including special spatial arrangements, new staff or student roles, and basic 
orqonizational characteristics. Behavioral features include chajiges in the 
btih.ivior or role interactions of all key actors, as identified by the developer. 
The final element, degree of knowledge about the objectives, content and philosophy 
of the implementation — will probably provide little implementation evaluation data 
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ul .'.n wjcji in ifiolation, since knowledge and bohaviors are not alv/ays related. 

When cievelopi;.:j a degree of imp lemon tation itistrument for the Individually 
Proscribed Insitruction (IPI) mathematics program, Evans and Shefflei. (1974) 
conc-».?ntratod on structural cind behavioral chciractoristics. However, l;.hey divided 
these charactorls tics into "instructional" and "organizational" categories. 

Many different approaches have been used in measuring implementation* In 
sunima^'izincj their examination of many studies involving degree of implementation 
measurements, Fullan and Pomfret (1975) concluded that participant or instructor 
percex:)tions v/ere inadequate means of assessing implementation. They further 
concluded questionnaire data and results of interviews with principals were of 
doubtful validity in assessing degree of implementation. Paul (1976) states: 

...factors for describing t:f fectiveness are difficult to measure. 
Studios of implementation which purport to measure chcinge outcomes 
should bo scrutinized to ascertain the method of measurement. 
Unless questionnaires have been validated with site visits and tocuu 
observations, their relicv.. Llity is doubtful. Observation using 
o-.^-e fully constructed checklists which focus on behavioral changes 
ift staff and students seems to be the most reliable method for 
measuring the extent of implementation. (p. 84) 

She Trier and Evans (1976) have identified some additional questions to 
consider in developing a degree of implementation instrument: 

1. V/ho determines the items? 

2. Who does the observing? 

3. How are observers trained? 

4. Ho-' many times a year should the checklist be used? 

In a 1976 address, Gephart identified four classes of problems which impede 
the measurement of degree of implementation: the purpose problem, the local 
adcipt mobility problem, the scaler problem and the innovation completion problem. 



A problem arises if the puriDose of the instruraent is not considered during 
dovelowiuent / or if cin already-developed instruraent is applied in a nev/ and 
d.\ 'tTerent setting. The purpose for which the instriunent is intended should 
deterniiae itr» characteristics. 

The second problem concerns the probability and accc^pteibili ty of local 
adaptations of an innovation. A degree of implementation instrument should be 
flexible enough to meD '.'.ire acceptability. Gephart states, "The problem becomes 
one of defining anticipated, actual, and appropriate adaptation" (p. 7). Minimal 
stcindards cind allowable variations should be noted by tlie measurement instrument. 

Since most innovations are rather complex, information gathered by a degree 
of implementation instrument is difficult to summarize. The scaler problem 
involves attempting to get a single score which adequately represents all 
components of an innovation. Gephart suggests that a profile of scores considering 
each of the various components "would be a more logical way of summarizing 
implementation than a single score" (p. 8) . 

The innovation completion problem is somewhat related to the local adapt— 
aViility problem. The transition from a functional, development-oriented setting 
into the user setting may expose gaps or weaknesses in an innovation which did 
no^. exist or were not relevant in the developmental setting. Some program 
v-^.iknesses may at first go unnoticed because of cin ideal setting at the develop- 
rr.^^ntal site, or local variation at the implementation site. The developer or 
local adopter c^m often "patch around" these weeiknesses. The innovation completion 
problem may also be the result of the "indispensible person phenomena." That is, 
th'j personal interest and involvement of the developer is an important program 
c.:.T.p '.nent which is impossible to transport to a new setting. Tliis factcr is 
e^:pecialLy significant if the developer is extensively involved xti the delivery 
of tile innovation in the original setting. 

6 

4 



C\;rt^iin miswcrc'Cptioas oxist: about: implomontation . First/ there is the 
"t-n-ichor-v r^<^^ system" iny th ' (Cophart , 1976): the belief that innovations such 
as [>i-ocjrrirn^; or mav.eri. .Is can bo ' s:^ fully developed that they Ccin be immediately 
i mp lemon t: .'id lu different classrooms with teachers of varying capability, 
,iriivolvom-::int and interest in the itinovation . (Gephart, 1976) The "universal 
adoption myth," imii)lies development of completely transportable and universally 
cipplicable innovations . 

Both myths are related to the Re search- Development-Diffusion (RDD) model 
dove lof led by Clark cxnd Guba <1972) . The MD model places responsibility for 
replication on the user, as though all components were totally acceptable. Of 
course/ tiiis is rarely the case, as evidenced by a later revision in the RDD 
model to reflect a configurational view (Guba and Clark/ 1974). 

Another misperception is the belief that the more faithful the replication 
effort to the original model, the more positive the t ntcome measures. This 
ii'irjlies that the hiqhest outcomes could be attained only by replications 

.'.nen'od in the same manner as the developmental or demonstration program 
cicjaLnst whicli degree of i in.olementation is being measured. This theory does not 
alL::w for enhancement by the replication sites or for local adaptation. 

Another misconception has been that continuous program fideJlty sliould be 
ericouragod, therefore/ instrumentation need measure degree of implementation at 
av.l/ one point in tine, without evaluating jngoing program changes. In reality 
programs usuall;-' chancvi after initial adoption and it's useful to monitor changes 
in v■^;•.Jrce of implementation- 
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:::it:|onQlG fo r Dev elopm ent of an Instrument to Measure Degree of Program Implementation 

The foregoing discussion suggests many consid-urations important in developing 
.1 .^ igreo of program implementation instrument. The instrument should be capable 
of assessing to what degree each prespecified program component has been implemented. 
Evviluators must v;ork closely v;ith d^^velopors to identify essential program 
components, and determine the best indicators lhat these components are being 
utilized properly. 

The instrument should be capable of generating a profile of program components/ 
rather than a single score which attempts to capsulize the innovation. Profiles , 
should allow for natural variation so that local sites car. adapt or implement a 
program at different stages. 

The instrument should also be 'able to measure program changes. Although 
most people realize that programs at replication sites evolve over time, few have 
considered that it is often impossible to adopt all features of a given program 
at the outset. Replication sites often initiate implementation with the most 
compatible, and easiest- to-implement components. Then, over time, components cem 
be added or refined. Tliis process affords replication sites immediate visibility 
with minimal local resistance to new procedures. However, unless the degree of 
implementation instrument is capable- of measuring these fine differences, it may 
b'i difficult to gain £m accurate picture of this gradual adoption strategy. 

The degree of implementation instrument should be reliable, but easy to 
aidminister. An on-site checklist requiring both observation and records 
examination is recommended. The checklist should cover both structural and 
behavioral program features. 

A final consideration concerns the use of data. In the authors' opinion, 
the belief that greater program fidelity will result in more positive student 
and program outcomes is fallacious. This myth assumes that tlie demonstration 
model being replica i'>::d is the ideal. In realit.y, no model is perfect; at best, 
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it in orily a w ' ::i.ncj guida based on procedures which v;erG dGmonstrably effective 
in one or more i^LloL testing sites. The re fore ^ there con be no assurcmce that 
high fidelity implementation will yield any better results than those obtained 
fiom a less faithful site. V/liaL con be hypothesized, however , is that high 
fidelity implementation will yiold student and program outcomes more consistent 
with those obtained in the prototype site(s). In addition, high fidelity 
implementation permits examination of relationships between particular program 
components and various student and program outcomes. 
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Development of Degree of Implementation Measures 
for an Experience Based Career Education Program 



This section will begin with a brief description of Experience-Based Career 
Education (EBCE) and the Nortliwest Regional Educational Laboratory's (NWREL) 



Education, or (CE)^. It will then cover the development of two degree of 
implementation instruments used with (CE) ^ and its replication sites, focusing 
on how results from those measures have been used. 



Experience-Based Career Education (EBCE) is a fundamentally different type 
of education for secondary students, X\fhile students in traditional high school 
programs attend classes all day, EBCE students spend a major portion of their 
time on learning projects in the community. EBCE activities are tailored to 
individual needs, abilities, learning styles and goals, ajid students are guided 
iu their learning by v/orking adults in the community. 

Tlirough their interactions with community members, EBCE students learn about 
careers, cibout life, about other people, abov.t themselves. They learn basic 
skills of critical thinking, science, personal and social development, functional 
citizenship and creative development- They gain competence in the skills adults 
need to function effectively in a technological society. They learn responsibility 
by helping design their o;^ learning activities, and by following a set of 
t-jrcountahility standards that parallel those standards working adults are expected 
to maintain on the joh. 

Perhaps most importcxnt, EBCE students learn how to learn: how to plan 
learning activities, how to find and use community resources, and how to build on 



demonstration project in Tigard, Oregon — Community Experiences for Career 



EBCE 
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exf» ' 'inuxi. 'i . :.v begin to see learning as a lifelong process with rewards 
directly r<.!latocl to each individual's personal goals. 

The \('E)n P rogram 

since the fall of 1972, a model EBCE program funded by the National Institute 
of Edueatiou (NIE) ajid sponsored by the Northwest Regional Educational Laboratory 
(MWREL) , has been operating in Tigard, Oregon.* The Tigard version of EBCE is a 
full-time educational alternative for high school juniors and seniors. The 
program servos about 10 percent of the eligible student body at Tigard High School. 

The maiority of student learning takes place at sites in the southwest 
Portland metropolitan area. t^en students are not pursuing community learning 
activities, their home base is the (CE)^ lecirning center. Center staff are not 
teachers in the traditional sense, but facilitators of student learning , who 
help students design and follow individualized 2 -earning plans within a prescribed 
curriculum. Volunteers at community sites serve major support roles in student 
lo.'' ruing. Policies for (CE)^ are detemined by a board of directors composed of 
students, paror.ts, employer?:, labor leaders and school district representatives. 
Vfiien students leave 2 ^^^^ receive a unique portfolio displaying their 

program experiences and accomplishments 7 and upon completion of program require^ 
ments they receive a standard diploma from Tigard High School. 

Pilot Sites 

Th-t NWREL EBCr-: orogrcim was cievciloped and refined over a three-year period at 
the (CE) demonstration site in Tigard, then tried out in four school districts 
in the Northwest. Each program was operated by the local district, with NWREL 

*EBC5 procjrams have also been developed, pilot tested and disseminated by tJie 
A^.^; ildchia Educational L£iboratory, Inc., in Charleston, West Virginia; Far VVest 
Lt^-»'>l:ato^y for Educational Pesearch cinJ Development in San Francisco, California; 
an.: Research for Better Schools, Inc. in Philadelphia, Pennsylvania. 
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proviclLncj training ioncl technical ar-jsistance / and pilot sites paying for program 
c ^r.;rationi5. During the 1976-77 school year, pilot sites are operational in 
Coi vi. I le , Washington; Hillsboro/ Oregon; Jefferson County, Colorado; Konnev/ick/ 
Washington; and Kodiak, Alaska. 

EBri^ Essential Characteristics Checklist^ 

Program developers and the evaluation staff felt that the essential 
characteristics of EBCE could be circumscribed within five descriptive components: 
1) is an individualized program; 2) EBCE is community-based; 3) EBCE is 

Gxpericnce-based, incorporating the daily activities of adults; 4) EBCE has its 
ov;n identity cind is comprehensive and integrated and 5) EBCE places major emphasis 
on students' career development. An Essential Characteristics Checklist was 
developed to measure the degree of implementation within each essential component 
area. Each comr-jonent comprises four to six essential characteristics. These 
characteristics are rated on a scale of 1 to 5, with prespecified anchor points. 

A background manual explains the purposes of the checklist, and describes 
special considerations for each essential characteristic. Anchor points have 
been set so that a 5 represents perfect prograin fidelity, while a 1 represents 
; an eintagonistic concept unacceptable within the framework of the NWREL EBCE model. 
Thus, a 2 would represent a highly questionable concept, a 3 a somewhat less 
questionable concept and a 4 an approximation of the intended concept, but still 
lackirrj perfect fidelity. Any rating below 4 indicates failure to implement an 
essential characteristic consistent with the NWREL-developed EBCE progrcim. 

During the past year, the Essential Characteristics Checklist was used in 
assessing the EBCE demonstration site (uhe original developmental site) and five 
pilot sites in the Morthwest; results among sites were compared. 

*Co:.-ie3 of this checklist can be obtained by writing the authors at NWREL, 710 
S.V;. Second Avenue, Portland, Oregon 97204. 
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I '.ring the proi-;ent year v/c plan to utilize the Esf^ontial Characteristics 
Ch.ecklii^t at each of 22 sites implcmonting the NWREL EBCE program under implementa- 
t i.on grants awarded through Part D of the Vocational Education Act.* This will 
a .Iov-; us to prepare a u-joful profile of coiriron implementation problems encountered 
by Part D site*-; acrosj; the country in their first year of operation. 

EBCE Process Checklist 

While the Essential Characteristics Checklist identifies basic policy cind 
philor«Oi)hical characteristics of an EBCE site, the Process Checklist is, designed 
to identify variations in procedures used to operate em EBCE program. Both were 
developed using the five EBCE handbooks developed by NWREL as a reference point. 
Tills process checklist consists of four sections: 1) EBCE objectives, 
2) mcinagement and organization processes, 3) curriculiam and instruction processes 
and 4) i^tudent service processes. Each section contains separate items — all of 
wliiwi'i focus on processes used- 

K>;; example, while 13 competencies were originally identified as important 
sujivival s-:ills for the Ticjard {C&:>2 students (such as "maintaining a checkbook"), 
the handbooks encourage the use of staff discussion and community input in 
s-■:lec^.ing competencies appropriate for a particular implementation site. Therefore, 
the prinary purpose of tlie Process Checklist is to describe exactly what processes 
are being used at any particular implementation site. 



*ThcsQ Pdrt D EBCE sites are located in Evergreen, Alabama/ Cordova, Alaska; 
Tucson, Arizona; Seacy , Arkansas; Denver, Lakewood, Eagle, Holyoke arid Rocky 
Ford, Colorado; Newark, Delaware; Hilo, Hawaii; Pocatello, Idaho; Decatur, 
Illinois; I^xington, Kentucky; Pontiac, Michigan; Chesterfield, Missouri; 
Grr:at Falls, Montana; Gsrxnd Island, Nebraska; Carson City, Nevad:; Minot, 
North Dakota; Medford, Oregon; Philadelphia, Pennsylvania; Waterirx.vn, South 
D r'^ta; San Antonio, Texas; Manassas and VJoodbridg^ , Virginia; B^llevue and 
i-ouv-tie, Ka^'^-'ngton; and 'Cheyenne, Glenrock, Lander and Laramie, Wyoming. 
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TiVi Hr.iiontlal Characteristics Checkli:;t and the Process; Checklist v/ere 



fiillocl out indiiipcndeatly at the becjinnincj and end of the 1974-75 school year by 
the pilot site project directors and the NV;ra^L ovaluator. Results of the first 
checklist adnini strati on were used to direct project staffs* attention tov/ard 
procjriim implementation issues that may have been neglected in the haste of getting 
students stsirted in program activities. The checklists also proved to be useful 
as a vehicle for defining Exi^erience- Based Career Education, £ind as ci congruency 
checklist for proposal developers seeking federal funds to implement an EBCE 



Utilization of EBCE Checklist Results 

Tiie most obvious application of any degree of implementation instrument is 
in assessing program fidelity. However, the authors have identified additional 
uses for the essential characteristics and process checklists used with the NWREL 
EBCE x)rogram. First, these instruments afford the potential adopter a good 
orientation to the characteristics' and processes of the EBCE program. In fact, 
uVi'REL EBCE Imx)lemen tatioii Technical Assistance staff have found these instruments 
to be effective technical assistance* devices . 

These instruments are also valuable in establishing initial priorities. 
Ir,i:\lemen tation sites which cannot immediately adopt all processes and character- 
istics might use these instruments to focus their initial implementation efforts 
ar.J gtiide future revisions. iMoreover, the instruments can be used to assess 
initial implementation efforts, to pinpoint weaknesses and to set the stage for 
future implementation. 

The instruments help stimulate discussions among site staff regarding program 
strengt:hr, and weaknesses. As a re:-*lt of such discussion staff may formalize 
plans to revise or complete program implementation. 



model. 
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Kinftlly/ th:} instruments could be used to evaluate program changes during the 
coiuT^e of a t^chool year or over several yoari;. NWi^L EBGE evaluation staff have 
fo\ii\ : tb,i£5 application UFsefu'l in identifying areas of implementation v/hich were 
incomplete at the beginning or middle of the year, but which had reached 
Ci:^:'nple tion by the end of the year. 

Conci U S ions 

In this paper we have attempted to sketch the research literature on 
mriasuring degree of program implementation and have identified some common myths 
in thi.: area. Vie then presented a rationale, description and uses for two 
measures dei^igned to assess the degree of implementation of the NWREL Experience- 
Based Career Education Program. Later this year we plan to administer the two 
E3CE implementation instruments to the Part D, VEA sites in 23 states who are 
using the NWi^L model of EBCE. Such data will provide the basis for an 
interesting analysis of the relationships among concept fidelity, process 
fidelity and outcomes. We recommend that other evaluators continue to examine 
t.\\a rationale for measur ■ of iTogram implementation and report new and varied 
u.ses for such measures. 
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